Add support for GCS emulators #177
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
manuteleco
commented
Jul 23, 2023
| @@ -18,7 +18,7 @@ gcs_upload( | |||
| upload_type = c("simple", "resumable") | |||
| ) | |||
|
|
|||
| gcs_upload_set_limit(upload_limit = 5000000L) | |||
| gcs_upload_set_limit(upload_limit = 5e+06) | |||
There was a problem hiding this comment.
I find this annoying. Not sure why roxygen2 decided to make this change.
|
Please do add a note to the NEWS.md and feel free to add as a contributor to DESCRIPTION - thank you! Looks like a useful addition. The rebuild of the documentation is usually done before CRAN launch |
Allow configuring a Google Cloud Storage emulator to be used instead of the real service. This can be achieved by providing a `STORAGE_EMULATOR_HOST` environment variable with the scheme, host and port of the emulator server (e.g.: http://localhost:8080). Besides using a different API endpoint, configuring an emulator also has the effect of lifting the requirement of providing valid authentication credentials. Emulators don't require or even implement authentication. This behavior is similar to the one implemented in official Google Cloud Storage client libraries developed and maintained by Google.
manuteleco
force-pushed
the
support_storage_emulator
branch
from
e52a37e
to
68c5019
Compare
|
Just in case you didn't get a notification from my last change, I have adjusted |
|
@manuteleco great thank you! I was on holiday |
|
@manuteleco I am getting some bug reports since this was merged, is it possible this is related? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Allow configuring a Google Cloud Storage emulator to be used instead of the real service. This can be achieved by providing a
STORAGE_EMULATOR_HOSTenvironment variable with the scheme, host and port of the emulator server (e.g.: http://localhost:8080).Besides using a different API endpoint, configuring an emulator also has the effect of lifting the requirement of providing valid authentication credentials. Emulators don't require or even implement authentication.
This behavior is similar to the one implemented in official Google Cloud Storage client libraries developed and maintained by Google.
Closes #176
It also has a weak dependency with MarkEdmondson1234/googleAuthR#228. Without that PR in
googleAuthR, using an emulator will still work ingoogleCloudStorageRbut it will unfortunately require valid credentials for the real GCS.Just like in #175 (comment), I wonder whether I have to build the
pkgdowndocumentation here or not. I also wonder what else should I include in this PR (e.g., an entry inNEWS.md, etc.).Also, be aware that I'm new to R, so feel free to point out any unidiomatic or plain odd code that needs to be fixed.
Finally, for testing I have used https://github.com/fsouza/fake-gcs-server, but it currently requires a few adjustments for it to work with
googleCloudStorageR: fsouza/fake-gcs-server#1258 and fsouza/fake-gcs-server#1252. With some luck, they will be accepted and merged eventually.I did not test every single GCS operation; it is possible some are still not working with an emulator yet. But I believe this PR already brings value even if it doesn't happen to provide 100% support. Any operation that is still not working could be refined later, once encountered.